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ELICIT  Overview 

Background 

As  part  of  its  network  eentric  warfare  initiative,  the  Command  and  Control  Researeh 
Program  (CCRP)  is  engaged  in  developing  and  testing  principles  of  organization  that 
significantly  transform  traditional  command  and  control  practices  that  are  hierarchical  in 
nature  and  transfer  power  and  decision  rights  to  the  edge  of  the  organization.  In  their 
book,  Power  to  the  Edge,  (2003)  Alberts  and  Hayes  argue  that  missions  designed  with 
superior  shared  awareness,  trust  and  self-synchronization  will  perform  with  greater  speed, 
precision,  effectiveness,  and  agility  than  missions  conducted  under  traditional 
hierarchical  command  structures. 

The  need  for  agility  in  information  Age  militaries  is  becoming  increasingly  important. 

As  discussed  in  Understanding  Command  and  Control  by  Alberts  and  Hayes  (2006),  in 
an  era  of  complex  coalition  civil-military  operations  understanding  how  to  organize  for 
agility  not  just  within  a  specific  organization  but  also  across  differing  organizations  and 
cultures  is  key. 

Historically  there  has  been  a  shortage  of  formal  experiment  data  on  the  efficacy  of  Edge 
vs.  traditional  organizational  styles.  Information  on  the  applicability  or  desirability  of  one 
organizational  style  vs.  another  has  been  anecdotal.  A  typical  battle  or  war  game  scenario 
has  too  many  uncontrolled  variables  to  produce  any  formal  experiment  data.  In  order  to 
enable  and  foster  the  conduct  of  formal  experiments  in  the  principles  of  organization,  the 
CCRP  created  an  ongoing  research  Project  called  EEICIT;  the  Experimental  Eaboratory 
for  Investigating  Collaboration,  Information-sharing  and  Trust.  As  part  of  this  program, 
the  CCRP  (Command  and  Control  Research  Program  www.dodccrp.org)  sponsored  the 
development  of  a  java-based  software  platform  that  can  be  used  to  run  multi-user 
experiments  in  information  sharing  to  empirically  explore  the  relationships  among 
approaches  to  command  and  control  and  organization,  team  and  individual 
characteristics,  as  well  as  value-related  measures  that  constitute  the  network-centric  value 
chain.  The  network-centric  value  chain  includes  a  robustly  networked  entity,  information 
and  collaboration  behaviors,  the  quality  of  information  and  awareness,  shared  awareness, 
synchronization,  and  effectiveness. 

The  CCRP  has  facilitated  the  formation  of  a  global  COl  (community  of  interest)  to 
accelerate  the  development  of  a  better  understanding  of  network-centric  concepts, 
approaches,  and  capabilities  as  well  as  power  to  the  edge  principles  by  building  upon 
each  others  work,  collaborating  on  campaigns  of  experimentation,  and  developing  a 
shared  body  of  knowledge.  This  paper  is  part  of  that  ongoing  effort. 

Experiment 

The  EEICIT  software  platform  currently  uses  a  scenario  that  focuses  on  the  task  of 
discovering  the  “who”,  “what”,  “when”,  and  “where”  of  a  future  terrorist  attack. 
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Information  is  provided  to  eaeh  of  the  participants  when  the  session  begins  and  at  two 
other  times  during  an  experiment  trial.  The  participants  receive  instructions  about  the 
nature  of  their  organization  and  the  information  sharing  modalities  available  to  them. 
These  constitute  an  experimental  treatment  and  can  be  varied  considerably  to  “create”  a 
particular  approach  to  command  and  control  and  organization.  The  participants  can  be 
selected  from  one  organization,  one  type  of  organization  or  from  many  organizations 
with  any  mix  of  individual  and  team  characteristics.  The  software  enables  numerous 
factors  to  be  independently  varied.  Transaction  logs  and  surveys  or  interviews  associated 
with  each  experiment  trial  collect  data  that  can  be  used  to  measure  information  sharing 
and  collaboration  behaviors  as  well  as  a  variety  of  value  metrics  including  the  ability  of 
individuals  and  teams  to  correctly  identify  the  “who”,  “what”,  “when”,  and  “where”  of  a 
future  terrorist  attack  and  the  time  required  to  do  so. 

Growing  Research  Community 

An  ELICIT  community  of  interest  (COl)  has  members  from  several  countries,  including 
Canada,  Germany,  Portugal,  Singapore,  UK  and  the  US,  who  are  actively  engaged  in 
experimentation  and  analysis  as  well  as  the  development  of  additional  tools  to  support 
this  research.  Both  civilian  and  military  institutions  are  involved. 

Experimentation,  research  and  analysis  have  been  ongoing.  Some  efforts  have  already 
resulted  in  research  papers  that  have  been  posted  on  the  ELICIT  website.  At  the  12* 
International  Command  and  Control  Research  and  Technology  Symposium,  several 
ELICIT -related  papers  were  presented.  One  of  these  won  the  Best  Student  Paper  Award. 
Eighteen  persons  from  six  countries  participated  in  the  post  ICCRTS  ELICIT  User  Group 
meeting.  Information  about  ELICIT,  the  COl  as  well  as  papers  related  to  ELICIT  can  be 
found  on  the  CCRP  Website,  http://www.dodccrp.org/html4/elicit.html 

ELICIT  Program  Resources 

The  initial  version  of  the  ELICIT  software  platform  has  been  and  continues  to  be 
enhanced.  This  paper  describes  two  significant  new  capabilities:  the  ability  to  configure 
organization  types  and  the  ability  to  support  agent-based  processing  that  have  been  added 
to  the  web-based  version  of  ELICIT  to  facilitate  constructive  experimentation. 

Several  organizations  have  also  developed  and  continue  to  develop  tools  to  automate  the 
extraction  of  useful  data  from  the  detailed  transaction  logs  created  by  the  ELICIT 
software.  In  addition,  an  experimental  campaign  has  been  defined  to  maximize  the  value 
of  coordinated  ELICIT  experimentation. 


New  Configurable  Organizations  Capability 

Motivation  For  Enhancement 

ELICIT  has  been  used  to  conduct  experiments  at  several  research  and  educational 
organizations  including  extensive  experiments  conducted  by  the  Naval  Post  Graduate 
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Sehool  Center  for  Edge  Performanee.  The  original  release  of  ELICIT  software  supported 
two  fixed  organizational  treatments:  Edge  and  C2  hierarehy. 


Figure  1  Edge  Organization 

The  Edge  organization,  as  shown  above  in  Eigure  1,  was  entirely  fiat  and  the  traditional 
C2  hierarehy  consisted  of  4  teams  of  4  persons  each  reporting  to  a  cross  team 
coordinator.  See  Eigure  2  below. 


Figure  2  C2  Hierarchy  Organization 

It  was  not  possible  to  vary  the  organization  structure  or  the  information  sharing  actions 
that  were  available  to  different  roles  in  an  organization.  For  example,  whether  a  given 
participant  could  send  information  (a  factoid  about  the  experiment  scenario)  to  another 
specified  participant.  Feedback  from  researches  in  multiple  organizations  and  countries 
indicated  that  it  would  be  helpful  to  be  able  to  define  experiment  groups  of  varying  sizes 
and  hierarchy  configurations  and  to  support  hybrid  organization  types. 

Enhancement  Capabilities 

The  new  configurable  organization  capability  provides  flexibility  in  the  following  areas: 

•  The  number  of  organization  types  has  been  expanded  to  as  many  as  the  researcher 
wishes  to  configure.  Previously  there  were  only  two  organization  types  (Edge  and 
C2) 

•  The  number  of  participants  in  an  organization  is  no  longer  restricted  to  17  players 
for  hierarchies 

•  The  number  of  levels  in  the  hierarchy  is  now  configurable.  Previously  it  was  one 
or  three 

•  The  names  of  the  roles  (titles)  for  each  person  are  now  configurable.  Previously 
these  were  hard  coded  for  each  of  the  two  defined  organization  types 
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•  The  partieipants  with  whom  a  given  partieipant  ean  Share  a  faetoid  is  now 
eontrolled  by  a  matrix  on  a  partieipant  to  partieipant  basis.  Previously  everyone 
eould  always  Share  with  everyone 

•  The  information  web  sites  that  a  partieipant  ean  Post  to  and  Pull  from  are  now 
eonfigurable.  Previously  there  was  only  one  option  for  eaeh  of  the  two 
organization  types  (Edge  and  C2) 

In  addition,  this  new  eapability  has  been  eonstructed  so  that  if  additional  partieipant 
aetions  were  added  to  EEICIT  in  the  future,  it  would  be  easy  to  make  them  eonfigurable 
by  organization  role.  If  for  example,  a  eapability  was  added  for  partieipants  to  send 
freeform  text  messages  in  the  future,  it  would  be  easy  to  make  this  new  eapability 
eonfigurable  based  on  a  partieipants  role  in  an  organization  so  that  it  eould  be  provided 
only  to  team  leaders,  or  only  to  edge  partieipants,  ete. 

The  EEICIT  web-based  software  platform  has  also  been  enhaneed  to  reeord  entries  to  the 
MyEaetoids  list  in  the  transaction  log  files.  This  capability  means  that  all  participant 
actions,  including  moving  an  item  to  a  personal  work  area,  are  now  recorded  in  the 
transaction  log. 

New  Avenues  For  Experimentation  Supported  By  Enhancement 

The  new  configurable  organizations  capability  greatly  expands  the  number  of 
experiments  that  can  be  performed  using  EEICIT. 

Eor  example,  in  the  original,  baseline  EEICIT  C2  structure,  only  the  cross-team 
coordinator  has  access  to  all  four  information  websites  (Who,  What,  Where  and  When). 
The  Who,  What,  Where  and  When  team  leaders  only  had  access  to  their  own  single 
respective  website.  Several  researches  were  interested  in  seeing  just  how  much 
expanding  this  access  to  each  of  the  four  team  leaders  improved  team  and  overall  group 
performance. 

Some  researches  want  to  be  able  to  mimic  existing  organization  styles  and  see  how 
participants  performed  in  their  own  vs.  alternate  organization  structures.  This  is  now 
possible.  Hybrid  organization  structures  can  now  be  constructed  in  EEICIT  to  see  which 
organization  elements  have  the  greatest  effects. 

In  addition,  it  should  be  pointed  out  that  the  factoid  distribution  continues  to  be 
configurable,  and  could  be  configured  in  conjunction  with  organization  roles  so  that 
certain  roles  received  better  or  worse,  or  more  or  less  information. 

The  result  of  this  effort  is  that  EEICIT  researchers  will  be  able  to  design  and  undertake 
numerous  additional  experiments.  See  Appendix  A  for  additional  information  on  the 
configurable  organizations  capability. 
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New  Agent-based  Processing  Capability 

Motivation  For  Enhancement 

Numerous  researehers  have  indieated  the  desirability  of  adding  the  ability  to  support 
Agent-based  proeessing.  That  is,  to  allow  software  agents  to  partieipate  in  ELICIT 
experiments.  Agent  partieipation  ean  be  of  two  types:  Hybrid  human  and  agent 
partieipant  experiments  and  agent-only  experiments.  Arranging  for  a  suitable  number  of 
qualified  human  partieipants  to  all  be  available  at  one  time,  is  often  one  of  the  most 
ehallenging  and  expensive  parts  of  executing  a  program  of  experimentation.  Erequently 
all  of  the  recruited  participants  are  not  actually  available  at  the  scheduled  time  due  to 
unforeseen  circumstances.  The  ability  to  have  an  agent  take  the  place  of  a  missing 
participant  greatly  increases  scheduling  flexibility. 

Agent-only  experiments  can  so  significantly  reduce  the  cost  of  running  experiments  that  a 
significantly  greater  number  of  experiments  can  be  run.  In  addition,  theories  of  behavior 
and  performance  could  be  validated  by  constructing  agents  with  specific  capabilities 
(personalities)  and  confirming  the  resulting  outcome  by  running  experiments  using  agents 
with  those  personalities.  Agent-based  experiments  are  not  intended  to  entirely  replace 
human  trials.  Insights  gleaned  from  agent-only  trials  will  need  to  be  replicated  with 
human  experiments  so  that  modeling  assumptions  do  not  effect  final  results. 

Enhancement  Capabilities 

The  web-based  version  of  ELICIT  has  been  enhanced  so  that  ELICIT  agents  can  be 
registered  for  an  experiment  trial,  either  in  combination  with  other  agents  or  with 
humans.  Everything  else  works  the  same.  The  humans  and  agents  communicate  with  the 
ELICIT  2.2  server  software  using  the  same  underlying  interface  and  have  all  the  same 
capabilities  so  that  a  human  participating  in  an  experiment  would  not  be  able  to  tell  if 
another  participant  were  a  human  or  an  agent. 

An  initial  stub  agent  has  been  created  to  validate  the  interface.  The  stub  agent  is  able  to 
execute  all  the  actions  that  a  human  can,  but  is  not  intelligent  about  how  and  when  it 
decides  to  take  these  actions.  An  effort  is  underway  to  develop  more  intelligent  agents 
that  model  human  decision  making  (within  the  scope  of  an  ELICIT  scenario)  and  human 
errors.  These  agents  will  be  able  to  formulate  Identify  messages  based  on  awareness  and 
understanding  of  the  factoids  to  which  they  have  been  exposed. 

New  Avenues  For  Experimentation  Supported  By  Enhancement 

The  new  agent  capability  greatly  expands  the  number  of  experiments  that  can  be  run  so 
that  more  variables  can  be  explored  and  a  sufficient  number  of  runs  can  be  conducted  for 
each  experimental  condition  so  that  results  will  be  statistically  significant. 

In  addition,  the  agent  capability  opens  new  avenues  for  exploring  theories  of  behavior 
and  performance.  Results  gleaned  by  human  participant  runs,  could  be  validated  by 
constructing  agents  with  specific  traits  and  confirming  the  resulting  outcome  of  running 
experiments  using  agents  with  those  traits,  matched  the  human  results. 
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Agents  eould  also  be  used  to  explore  the  effeet  of  eertain  “personality  traits”  on  group 
performanee.  For  example,  how  many  persons  with  the  “information  hoarding”  trait  does 
it  take  to  negate  a  group’s  performanee?  Sueh  experiments  eould  help  to  quantify  and 
validate  rich  targets  for  team  training.  See  Appendix  B  for  more  information  on  the 
ELICIT  agent-based  processing  capabilities. 

The  agent  capability  complements  and  supplements  ELICIT  experiments  done  with 
human  participants.  The  agent  capability  is  not  intended  to  replace  experiments  with 
human  subjects,  and  promising  results  from  experiments  with  agents  will  still  need  to  be 
validated  with  humans.  Replication  of  the  human  baseline  ELICIT  experiments  is  still  a 
requirements  for  joining  the  ELICIT  community  of  interest.  Some  of  the  “softer”  aspects 
of  ELICIT  experimentation,  particularly  sharing  of  free-form  text  information  via  post¬ 
cards  or  an  external  chat  system  are  outside  the  scope  of  the  current  agent  development 
effort  to  create  moderately  intelligent  agents  that  are  configurable.  Therefore  these 
avenues  of  research  will  remain  the  purview  of  human  subjects  for  some  time.  The  lower 
priority  on  creating  agents  that  can  handle  sharing  of  free-form  ELICIT  data  is  due  to  the 
difficulty  of  the  task.  It  is  not  for  lack  of  interest;  as  the  effects  of  sharing  freeform 
information  (that  could  represent  requests  for  specific  information,  sharing  of  theories 
about  the  task  situation,  or  suggestions  for  modes  of  self  organization)  are  a  very  fertile 
area  for  ELICIT  research.  As  more  subtle  aspects  of  sharing,  trust  and  awareness  are 
studied,  there  will  be  an  on  going  tension  between  the  models  instantiated  in  the  agent 
implementations  and  the  designs  of  the  experiments  that  use  them. 

Now  that  we  have  greatly  expanded  the  universe  of  possible  experiments  that  can  be  run 
with  ELICIT,  the  next  logical  step  is  to  coordinate  these  efforts  into  an  overall  campaign 
of  experimentation. 


Experimentation  Campaign 

The  Command  and  Control  (C2)  Community  has  advanced  in  its  conduct  of  research  to 
the  point  of  organizing  campaigns  of  experiments  oriented  toward  a  common  set  of 
objectives  (Alberts  and  Hayes  2005).  The  idea  is  for  each  experiment  to  both  build  upon 
and  extend  those  that  have  been  conducted  previously,  and  for  the  cumulative  new 
knowledge  generated  through  such  experiments  to  combine  and  to  address  important  C2 
issues  through  combination. 

Despite  this  progress,  however,  designing  an  experimentation  campaign  can  be  a 
challenging  endeavor.  It  is  unclear  often  which  key  dimensions  and  variables  offer  the 
greatest  potential  to  contribute  new  knowledge  through  experimentation,  and  without 
some  coherent  framework  for  organizing  and  interrelating  the  various  experiments, 
planning  a  sequence  of  experimentation  designs  and  sessions  can  devolve  into  a 
haphazard  activity.  Driven  largely  through  our  ongoing  ELICIT  work,  we  have  drawn 
from  the  academic  and  practitioner  literatures  to  develop  such  a  coherent  framework  for 
organizing  and  interrelating  the  various  experiments.  This  framework  is  comprised  of 
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four  dimensions:  1)  level  of  analysis,  2)  C2  approach,  3)  measure  of  effectiveness,  and  4) 
research  method.  We  summarize  these  dimensions  in  Table  1,  and  discuss  each  in  turn. 
To  enhance  continuity,  this  discussion  is  kept  to  a  relatively  high  level.  We  elaborate 
details  associated  with  the  experimentation  campaign  in  Appendix  C  for  reference. 

Multidimensional  Framework 

Table  1  Campaign  Dimensions  (adapted  fromNissen  2007) 


Campaign  Dimensions 


Level  of 

Analysis^ 

Independent^ 

(C2  Approach) 

Dependent^ 

(Measure) 

Research"^ 

(Method) 

Performance 

Analytical 

Social 

Organization 

Risk 

Computational 

Cognitive 

Process 

Agility 

Experimental 

Information 

People 

Learning 

Field 

Physical 

Technology 

Change 

Action 

1  Alberts  et  al.  (2001 )  2  Leavitt  (1 965)  3  CCRP  workshop  (2007)  4  Nissen  &  Buettner  (2004) 
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The  level  of  analysis  dimension  derives  from  Alberts  et  al.  (2001),  and  includes 
four  levels  relevant  to  C2  research:  1)  physical,  2)  information,  3)  cognitive  and  4)  social. 
The  C2  approach  dimension  derives  from  Leavitt  (1965),  and  can  be  thought  fruitfully  as 
involving  the  independent  variables  that  are  selected  for  analysis  via  an  experiment. 
Leavitt  discusses  four  key  aspects  of  organizational  performance:  1)  organization 
structures,  2)  work  processes,  3)  people  and  4)  technologies.  The  measure  of 
effectiveness  dimension  derives  from  CCRP  (2007),  and  can  be  thought  fruitfully  as 
involving  the  dependent  variables  that  are  selected  for  analysis  via  an  experiment.  The 
list  of  measures  can  be  very  long,  but  we  have  identified  a  set  of  five  that  capture  the 
most  pressing  aspects  of  C2  effectiveness  today:  1)  performance,  2)  risk,  3)  agility,  4) 
learning  and  5)  change.  Research  method  derives  from  Nissen  and  Buettner  (2004),  and 
refers  to  the  approach  that  an  experimenter  will  take  to  developing  new  C2  knowledge. 
This  dimension  lies  somewhat  outside  of  the  framework  for  describing  experimentation 
campaigns,  for  laboratory  experimentation  represents  one  of  several  alternate  research 
methods  that  can  be  employed.  The  five  methods  associated  with  this  framework  are 
interrelated  closely,  however,  in  that  they  form  something  of  a  sequence  or  life  cycle  that 
can  be  used  for  planning  a  research  campaign:  1)  analytical  methods,  2)  computational 
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methods,  3)  laboratory  experimentation,  4)  field  research  and  5)  action  research; 
theorizing,  which  begins  many  research  campaigns,  can  be  considered  as  a  sixth 
“method,”,  but  much  of  the  theorizing  taking  place  currently  in  the  C2  domain  does  not 
appear  to  be  guided  methodologically.  Each  research  method  has  distinct  advantages  and 
disadvantages  with  respect  to  others.  Each  of  the  four  levels  of  analysis,  four  C2 
approaches,  five  measures  of  effectiveness,  and  five  research  methods  is  described  in 
Appendix  C. 

Campaign  Space  Visualization 


Campaign  Space  Viz 


Measure  of 
Effectiveness 


80  unique  study  areas  (4  x  4  x  5  =  80)  x  5  methods 

Figure  3  Campaign  Space  Visualization  (adapted  from  Nissen  2007) 

Additionally,  we  can  combine  these  dimensions  in  a  manner  that  enables  us  to 
visualize  their  interrelations,  and  to  visualize  the  campaign  space  as  a  whole.  Eigure  3 
depicts  the  campaign  space  dimensions  as  such.  Here  we  have  the  first  three  dimensions 
(i.e.,  level  of  analysis,  C2  approach,  measure  of  effectiveness)  arranged  into  a  three- 
dimensional  coordinate  space.  Because  visualization  beyond  three  dimensions  is  difficult 
for  most  people,  we  list  research  method  outside  of  this  space,  and  depict  the  different 
methods  by  using  different  colors.  Eor  instance,  analytical  methods  are  colored  blue, 
computational  methods  are  green,  experimentation  methods  are  orange,  field  methods  are 
red,  and  action  methods  are  purple.  With  this,  any  particular  research  project  can  be 
viewed  in  relation  to  others  within  a  campaign  to  assess  their  combined  coverage  and 
interrelations  in  the  campaign  space.  Where  all  such  projects  tend  to  concentrate  in  one  or 
a  small  number  of  areas  within  this  space  (e.g.,  weighted  heavily  toward  new 
technologies),  one  could  see  quickly  that  other  projects  would  need  to  be  planned  to 
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cover  additional  areas.  Ideally,  one  would  like  to  see  experiments  fill  this  space  broadly 
through  a  campaign,  for  it  is  difficult  with  research  to  tell  where  the  high  payoff 
experiments  will  lie.  However,  where  research  results  in  one  or  more  specific  areas  prove 
to  have  high  payoffs  in  terms  of  new  knowledge  generated,  one  could  see  quickly  that 
others  in  these  areas  might  be  warranted,  but  in  particular  that  other  research  methods 
(esp.  field  research)  corresponding  to  these  same  areas  would  be  in  order. 

Example  Population  with  Recent  Studies 


Campaign  Space  Viz 


Measure  of 
Effectiveness 


Figure  4  Populated  Campaign  Space  (adapted  from  Nissen  2007) 

In  Figure  4  we  populate  this  space  with  a  select  set  of  recent  published  C2 
investigations,  (see  Nissen  2005,  Orr  and  Nissen  2006,  Ruddy  2007,  Leweling  and 
Nissen  2007,  Gateau  et  al.  2007,  Looney  and  Nissen  2006).  Each  study  is  listed  and  color 
coded  on  the  bottom-right  of  the  figure,  and  a  capital  letter  is  placed  within  the  campaign 
space  to  depict  where  such  study  would  be  located.  For  instance,  the  letter  “A”  represents 
the  (green)  computational  method  employed  in  the  Nissen  (2005)  study,  and  appears  at 
two  different  coordinates  within  the  space;  1)  social  level  of  analysis,  organization  C2 
approach,  and  performance  measure  of  effectiveness;  and  2)  social  level  of  analysis, 
organization  C2  approach,  and  risk  measure  of  effectiveness.  Notice  that  this  same  study 
examines  two  measures  of  effectiveness:  performance  and  risk.  Letters  representing  the 
other  studies  are  plotted  in  the  same  manner.  Notice  that  these  studies  tend  to  cluster  at 
the  social  level  of  analysis,  organization  C2  approach,  and  performance  measure  of 
effectiveness,  but  that  they  span  the  range  of  C2  approach  and  measure  of  effectiveness. 
Clearly  there  are  many  other  areas  within  this  campaign  space  that  would  appear  to  merit 
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research  attention.  Being  able  to  visualize  such  areas  represents  a  strength  of  this 
technique. 

Self-Organization  and  Self-Synchronization 

Despite  the  connotations  of  an  “experimentation  campaign”  that  suggest  a 
centrally  planned  and  controlled  endeavor,  we  have  just  the  opposite  in  mind.  Drawing 
from  the  same  Edge  organizing  principles  (e.g.,  see  Alberts  and  Hayes  2003)  that  drive 
much  of  this  research,  we  wish  instead  to  establish  sufficient  goals  and  conditions  for 
researchers  to  decide  among  themselves  who  will  undertake  which  projects,  and  in  which 
order,  and  to  synchronize  their  own  activities  with  those  of  other  researchers  to  ensure 
that  research  projects  fill  the  Campaign  Space  in  a  productive  and  efficient  manner. 

A  clear  first  step  toward  establishing  such  goals  and  conditions  takes  place 
through  this  article  of  course,  along  with  the  formal  and  informal  discussions  likely  to 
ensue,  as  we  lay  out  the  space  for  researchers  to  consider  as  they  plan  their  projects. 
However,  as  a  community,  we  will  need  to  do  more.  Three  logical  next  steps  follow.  Eirst 
would  be  to  populate  the  Campaign  Space  with  studies  that  have  been  accomplished  to 
date.  Such  population  could  follow  the  example  delineated  in  the  figure  above,  but  would 
draw  from  a  much  broader  sample  of  relevant  studies.  Second  would  be  to  annotate  the 
Campaign  Space,  or  document  separately  the  key  conclusions  and  implications  of  each 
project,  with  the  objective  of  using  such  metaresearch  to  identify  the  most  promising 
avenues  for  continued  research. 

Third  would  be  to  identify  and  interact  with  a  relatively  large  number  of 
researchers,  using  such  interaction  to  share  the  ongoing  results  of  this  metaresearch  (esp. 
sharing  the  Campaign  Space  as  it  continues  to  develop,  and  disseminating  the  most 
promising  avenues  for  continued  research.),  and  to  track  who  is  doing  research  in  which 
areas.  Eourth  would  be  to  continue  this  pattern  of  populating  the  Campaign  Space, 
sharing  metaresearch  results,  and  guiding  self-organizing  and  self-synchronizing  research 
projects.  Although  such  self-organization  and  self-synchronization  would  be  guided  by  a 
common  Campaign  Space,  and  would  be  prompted  by  periodic  research  sharing  and 
interaction,  individual  researchers  themselves  would  be  responsible  for  their  collective 
organization  and  synchronization. 

Hence,  we  would  use  an  Edge  approach  to  organizing  Edge  research,  and  we 
could  study  our  own  research  organization  to  glean  insights  into  organizing  other  kinds  of 
activities  and  endeavors  (e.g.,  military  engagement,  disaster  relief,  coalition  operations) 
via  Edge  principles.  This  kind  of  autoresearch,  and  the  metaresearch  noted  above,  offers 
great  promise,  but  will  require  sustained  effort  over  a  considerable  period  of  time.  We 
hope  that  the  Campaign  Space  outlined  here  will  serve  as  a  metaphorical  cornerstone  for 
building  an  impressive  knowledge  structure  both  cumulatively  and  longitudinally. 
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Conclusion  and  Suggestions  for  Future  Research 

ELICIT  is  the  Experimental  Laboratory  for  Investigating  Information-sharing 
Collaboration  and  Trust.  A  projeet  of  the  Command  and  Control  Researeh  Program,  the 
ELICIT  project  developed  an  online  multi-user  software  platform  for  conducting 
experiments  in  information-sharing  and  trust.  The  initial  version  of  the  software  allowed 
researchers  to  model  and  experiment  with  a  limited  set  of  C2  organizations,  processes 
and  approaches  in  a  computer-instrumented  environment.  The  ELICIT  software  has  since 
been  enhanced  to  allow  organization  type  to  be  configurable,  thereby  allowing 
researchers  to  experiment  with  a  wider  variety  of  C2  organizations,  processes  and 
approaches.  This  facilitates  greater  use  of  laboratory  experimentation  to  examine  a  wider 
range  of  theoretical  propositions  and  hypotheses  empirically.  In  particular,  the 
performance  of  many  hybrid  and  blended  organizational  forms  can  be  assessed 
experimentally  now  via  ELICIT,  and  experimentation  can  be  employed  to  augment  other 
research  methods.  Eor  instance,  the  empirical  comparison  of  multiple,  theoretical, 
archetypal  organizational  forms  by  Gateau  et  al.  (2007)  can  be  examined  now  in  the 
laboratory  using  ELICIT.  This  represents  a  very  promising,  near-term  topic  of  future 
research. 

In  addition,  the  software  has  been  further  enhanced  to  support  software  agents  as  well  as 
human  participants,  greatly  expanding  the  avenues  for  research.  Eor  instance,  validational 
and  exploratory  research  are  needed  to  examine  the  relative  fidelity  and  behavior  of  agent 
players  with  respect  to  their  human  counterparts,  and  studies  to  examine  the  relative 
strengths  and  weaknesses  of  human  and  software  agents  is  warranted.  In  particular, 
researchers  may  find  it  useful  to  build  upon  experimentation  work  by  Nissen  and 
Sengupta  (2006)  on  human  and  software  agents  in  the  supply  chain,  and  to  adapt  their 
findings  regarding  human-software  agent  integration  to  the  C2  domain. 

Eurther,  although  the  introduction  and  use  of  ELICIT  remain  relatively  recent  events, 
considerable  research  has  been  conducted  already  using  this  experimentation  platform, 
and  the  C2  Research  Community  is  gaining  commensurate  experience  and  insight  into 
sound  research  design.  Eor  instance,  the  research  design  articulated  by  Leweling  and 
Nissen  (2007)  can  serve  well  to  guide  a  diversity  of  follow-on  laboratory 
experimentation.  Building  upon  such  research  and  experience,  we  develop  a 
multidimensional  campaign  for  continued  experimentation  using  the  ELICIT  platform. 
The  campaign  is  populated  with  recent  studies  and  guides  future  researchers  toward  high- 
payoff  research  areas  that  can  be  addressed  using  ELICIT. 

In  particular,  research  addressing  the  cognitive  and  social  levels  of  analysis  appears  to  be 
especially  promising  at  this  time,  for  it  balances  the  tremendous  effort  being  expended  to 
study  the  information  and  physical  levels.  Moreover,  research  addressing  the  effects  of 
manipulating  all  four  C2  approaches  (i.e.,  organization,  process,  people,  technology) — 
both  individually  and  in  combination — appears  to  be  especially  promising  at  this  time 
also,  for  it  balances  the  tremendous  effort  being  expended  to  study  purely  technological 
approaches.  Examining  multiple  measures  (i.e.,  in  addition  to  performance)  offers  great 
merit  too,  as  alternate  dependent  measures  offer  tradeoff  spaces  for  leaders  and  decision 
makers  to  consider.  Tradeoffs  between  performance  (e.g.,  in  terms  of  mission  speed)  and 
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risk  or  agility,  for  instanee,  are  omnipresent  in  C2,  but  few  studies  to  date  have  examined 
the  underlying  tension,  or  have  offered  praetieal  policy  and  decision-making  guidance. 
Finally,  something  of  a  life  cycle  of  research  methods  can  be  seen  in  the  campaign.  One 
may  find  it  productive  generally  to  begin  a  campaign  with  analytical  methods,  and  then 
use  computational  tools  to  examine  the  most  promising  ideas.  Such  ideas  can  be  tested 
further  via  laboratory  experimentation,  after  which  only  the  most  promising  approaches 
can  be  assessed  in  the  field,  and  implemented  ultimately  via  action  research.  Together, 
the  various  dimensions — and  their  interrelations — of  the  Campaign  Space  offer  excellent 
potential  to  guide  and  integrate  a  great  many,  diverse  studies  into  a  coherent  C2  research 
stream.  This  is  the  idea  behind  a  campaign,  and  we  look  forward  both  to  contributing 
directly  to  and  helping  others  in  this  campaign. 
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Appendix  A  -  ELICIT  Configurable  Organization  Capability  Details 

ELICIT  Organizations  are  no  longer  restricted  to  two  hard  coded  organization  types. 
Now,  a  variety  of  organization  types  can  be  configured  using  simple  organizationtype.csv 
text  files. 


Configuration  Dashboard  Console  Logs  [  Config  Files  |  Agent  Registration 

Configuration  fiies 

Select  configuration  file  to  upload  onto  server. 

File  name  should  begin  with  factoidset,  names,  countries  or  organization  and  ends  with  .csv 
ELICIT  Configuration  Agent  Configuration  Agent  JAR  File 

@  o  o 


File  name  I 


Browse.. 


Upload  1 1  Reset 


□ 


factoidsetl-17.csv 
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□ 
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□ 
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Factoid  Sets 

^James 

Countries 

Orq 

anizations 

Aaent  Conf  Files 

Aaent  JAR  Files 

ibq 

I'ul 

i>.-l 

l).d 

1  '.-1 

□ 

factoidset5-17.csv 

□ 

namesl7.csv 

□ 

countriesl.csv 

□ 

organi2ationTypeC2-17.csv 

□ 

WhoAgent.csv 

□ 

WhoAqent.iari 

□ 

factoidsetS-S.csv 

□ 

countries4,csv 

□ 

organizationE C2 mixed-17.csv 

□ 

WhoAgent2,csv 

□ 

factoidset4-8.c5v 

□ 

countriesS.csv 

□ 

organizationE-17.csv 

□ 

factoidset4-l.csv 

□ 

countriesZ.csv 

factoidset4-17.csv 


factoidset3-17.csv 


factoidsetl'S.csv 


factoidset4-2.csv 


factoidsetl-l.csv 


factoidset4-5.csv 


factoidset2-17.csv 


iFactoidset  Files  (12)|Name  Files  (i)  ICountries  Files  (4)|Orqanization  Files  (3) 


lAqent  Conf  Files  (2)|Aaent  JAR  Files  (1)1 


Figure  5  ELICIT  Configuration  Screen  For  Loading  Organization  Type  Files, 

Figure  5  shows  how  an  organization  type. csv  file  can  be  loaded  into  the  ELICIT  server 
by  a  researcher.  Support  from  an  information  technology  specialist  is  not  required. 

The  file  is  in  .csv  (comma-delimited)  format,  which  means  that  the  value  in  each  field 
is  separated  with  a  comma  or  similar  marker.  In  the  ELICIT  configuration  files,  the  fields 
are  separated  with  vertical  bar  (|).  Figure  6  below  contains  an  example 
organizationtype.csv  file. 

A  key  at  the  top  of  the  file  explains  data  in  the  file. 

n  I  Role  I  team  I  Country  1 1 1  2  I  3  I  ...17  I  Web  sitel  |  Web  site2  |  Web  site3  |  Web  site4 

In  the  actual  table  that  begins  on  line  5,  you  see  a  player  number,  a  team  member  identity, 
a  country  label  for  that  player  (if  a  code  of  <country*>  is  supplied,  then  the  nth  entry  in 
the  country  table  specified  for  the  experiment  trial  is  used)  and  a  series  of  numeral  1  ’s 
with  a  single  0,  which  is  sequentially  arrayed  across  the  grid.  If  there  is  a  1  in  the  first 
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player  position,  then  the  player  assoeiated  with  the  row  ean  share  with  the  first  player.  If 
there  is  a  0  in  the  5**'  player  position,  than  the  player  assoeiated  with  the  row  eannot  share 
with  the  5**'  player.  If  there  is  a  I  in  a  Web  site  eolumn,  then  the  player  assoeiated  with 
that  row  ean  access  the  Web  site.  If  there  is  a  0  in  a  Web  site  column,  then  the  player 
associated  with  that  row  cannot  access  the  Web  site. 

In  the  following  example,  the  file  is  organizationTypeC2-17.csv  (the  17-player 
configuration  file  for  a  C2  organization).  In  this  organization  type,  the  Cross-Team 
Coordinator  and  four  Team  Leaders  (who  coordinate  who,  what,  when,  and  where 
information),  have  different  access  privileges  to  the  Web  sites.  The  Cross-Team 
Coordinator  can  access  all  four  Web  sites.  The  remaining  16  players  each  can  access  only 
one  Web  site.  Line  22  assigns  these  players  to  group  B,  the  C2  organization.  The  final 
lines  list  the  names  of  the  Web  sites. 


How  to  read  table: 

n I  Role  I TeamICountry I  1 |2|3|4|5|6|7|8|9|10|11|12|13|14|15|16| 17 | websitel | 
website2 | website3 | website4 | 

<begin  actual  table> 

1 1  Cross  Team  Coordinator ||0|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1| 

2  I  Team  leader 

I  Who  I <countryl> |1|0|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|0|0|0| 

3  I  Team  leader 

I  what  I <country2> |1|1|0|1|1|1|1|1|1|1|1|1|1|1|1|1|1|0|1|0|0| 

4  I  Team  leader 

I  where | <country3> |1|1|1|0|1|1|1|1|1|1|1|1|1|1|1|1|1|0|0|1|0| 

5  I  Team  leader 

I  when  I <country4 |1|1|1|1|0|1|1|1|1|1|1|1|1|1|1|1|1|0|0|0|1| 

6  I  Team  member 

I  Who  I <countryl> |1|1|1|1|1|0|1|1|1|1|1|1|1|1|1|1|1|1|0|0|0| 

7  I  Team  member 

I  Who  I <countryl> |1|1|1|1|1|1|0|1|1|1|1|1|1|1|1|1|1|1|0|0|0| 

8  I  Team  member 

I  who  I <countryl> |1|1|1|1|1|1|1|0|1|1|1|1|1|1|1|1|1|1|0|0|0| 

9  I  Team  member 

I  what  I <country2> |1|1|1|1|1|1|1|1|0|1|1|1|1|1|1|1|1|0|1|0|0| 

10  I  Team  member 

I  what  I <country2> |1|1|1|1|1|1|1|1|1|0|1|1|1|1|1|1|1|0|1|0|0| 

11  I  Team  member 

I  what  I <country2> |1|1|1|1|1|1|1|1|1|1|0|1|1|1|1|1|1|0|1|0|0| 

12  I  Team  member 

I  where | <country3> |1|1|1|1|1|1|1|1|1|1|1|0|1|1|1|1|1|0|0|1|0| 

13  I  Team  member 

I  where | <country3> |1|1|1|1|1|1|1|1|1|1|1|1|0|1|1|1|1|0|0|1|0| 

14  I  Team  member 

I  where | <country3> |1|1|1|1|1|1|1|1|1|1|1|1|1|0|1|1|1|0|0|1|0| 

15  I  Team  member 

I  when  I <country4> |1|1|1|1|1|1|1|1|1|1|1|1|1|1|0|1|1|0|0|0|1| 

16  I  Team  member 

I  when  I <ountry4> |1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|0|1|0|0|0|1| 

17  I  Team  member 

I  when  I <country4> |1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|0|0|0|0|1| 
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B 

Who 

what 

where 

when 

<end  actual  table> 

Figure  6  The  OrganizationC2-17.csv  File  For  The  Baseline  C2  Organization 
Scenario 
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Appendix  B  -  ELICIT  Agent-based  Processing  Capability  Details 

The  ELICIT  web-based  software  has  been  enhanced  so  that  it  can  recognize  and  run 
agent-based  processes  either  as  part  of  agent-only  experiment  runs  or  as  part  of  hybrid 
human  and  agent-based  participant  experiments. 

The  following  steps  should  be  done  to  develop  and  deploy  a  new  agent; 

1)  Implement  interface  Agentifc  or  extend  AbstractAgent  software  (create  the  agent 
software  program) 

2)  Create  an  agent  configuration  file  (.csv)  to  contain  the  parameters  driving  that 
agent  software 

3)  Deploy  agent  library  jar  file  and  agent  configuration  file  to  the  ELICIT  server 
using  the  “Configuration  files”  page. 


Any  agent  implementation  (agent  configuration  file  and  agent  jar  library  file)  may  be 
uploaded  to  the  ELICIT  server  on  the  “Configuration  files”  page. 


Configuration  Dashboard  Console  Logs  [  Config  Files  |  Agent  Registration 

Configuration  files 

Select  configuration  file  to  upload  onto  server. 

File  name  should  begin  with  factoidset,  names,  countries  or  organization  and  ends  with  .csv 

ELICIT  Configuration  Agent  Configuration  Agent  JAR  File 

®  o _  o 

File  name 


Browse.. 


Upload  1  [  Reset 


□ 


factoidsetl-17.csv 


□ 


□ 


□ 


□ 


□ 


□ 


□ 


Factoid  Sets 

Sames 

Countries 

Orq 

anizations 

Aqent  Conf  Files 

Aqe 

nt  JAR  Files 

Did 

I'.-i 

l).-I 

Pi-1 

□ 

factoidset5-17.csv 

□ 

namesl7.csv 

□ 

countriesl.csv 

□ 

organizationTypeC2-17.csv 

□ 

WhoAgent.csv 

□ 

vVhoAgent.jarj 

□ 

factoidsetS-S.csv 

□ 

countries4.csv 

□ 

organi2ationE_C2_mixed-17.csv  □ 

WhoAgent2.csv 

□ 

factoidset4-8.csv 

□ 

countriesS.csv 

□ 

organt2ationE-17.csv 

□ 

factoidset4-l.csv 

□ 

countries2.csv 

factoidset4-17.csv 


factoidset3-17.csv 


factoidsetl-S.csv 


factoidset4-2.csv 


factoidsetl-l.csv 


factoidset4-5.csv 


factoidset2-17.csv 


^actoidset  Files  (12)|Name  Files  (1)  [Countries  Files  (4)|Orqanization  Files  (3) 


lAqent  Conf  Files  (2)|Aqent  JAR  Files  (1)1 


Figure  7  ELICIT  Configuration  Screen  for  Loading  the  Agent-based  Processing 
Files 


After  the  agent  software  and  .csv  configuration  files  have  been  uploaded,  it  is  possible  to 
add/register  a  new  agent  in  the  ELICIT  application  so  that  it  can  participant  in  an 
experiment. 
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1)  Cliek  on  the  “Agent  Registration”  option  on  the  Server  Moderator  Configuration  page; 


Configuration  Dashboard  Console  Logs  Config  Files  Agent  Regist'ation 


2)  Select  the  agent  configuration  file  on  the  “Agent  Registration”  page: 


Configuration  Dashboard  Console  Logs  Config  Files 


Agent  Registration 


Software 

None 

V  i 

None 

WhoAgent.csv 

WhoAqent2.csv 

Figure  8  ELICIT  Agent  Registration  Screen 

The  dropdown  is  populated  by  the  values  of  the  names  of  agent  configuration  files  that 
were  uploaded  to  the  server  via  the  Configuration  page.  Each  agent  configuration  file 
references  a  specific  agent  library  jar  file  (specific  agent  software  file).  A  single  agent 
.CSV  file  can  be  registered  multiple  times  (to  represent  multiple  participants)  in  an 
experiment  trial. 

Eor  example,  the  WhoAgent2.csv  file  has  the  following  content: 

Agent's  actions  triggers: 

Click  Ready  button  after  configured  interval  after  a  trial  set  is  started.  Action  is 
triggered  by  receipt  of  a  New  Trial  Set  instruction  URL. 

Post  factoid  to  who  website  after  configurable  delay  factor.  Action  is  triggered  by 
receipt  of  a  new  factoid  in  the  inbox. 

Pull  at  configurable  interval  from  a  specified  website  (Who). 

Share  factoid  after  configured  interval  after  factoid  is  received  in  inbox.  Share  with  first 
participant  on  Share  list. 

Move  factoid  to  MyFactoids  after  configured  interval  after  factoid  is  received  in  inbox. 

Identify  action  after  configured  internal  x  1 0  after  factoid  is  received  in  inbox.  Identify 
content  is  hard  coded  (only  three  fields  are  provide). 

Once  received  by  the  Server  via  the  Agent  API,  the  server  treats  these  actions  as  if  they 
were  generated  by  a  human.  Player. 

<begin  agent  configuration  parameters> 

WhoAgent.jar 

net.parityinc.ccrp.web.agent.impl.Agentl 
ready IntervalDelay|Time  interval  to  click  Ready  button]  1000 
postToWhoSiteDelayjTime  interval  to  post  factoid  to  WHO  website]  10000 
pullFromWhoSiteDelayjTime  interval  to  pull  from  WHO  website]  10000 
shareWithFirstDelayjTime  interval  to  share  with  first  participant  on  Share  list]  10000 
moveToMyFactoidsDelay]Time  interval  to  move  factoid  to  MyFactoids]  10000 
identify ActionDelay]Time  interval  to  identify  action]  10000 
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Onee  an  agent  has  been  registered  to  the  server,  the  agent  will  appear  on  the  Server 
dashboard  screen.  Note  that  the  content  of  the  dashboard  fields  varies  slightly  between 
human  and  agent  participants. 


Configuration  Dashboard  Console  Logs  Config  Files  Agent  Registration 


l.'.-l 

Num 

Host 

SessionID 

Name 

Connect 

Status 

Ready 

Client 

State 

Num 

Identifies 

Text  of  last  identify 

Roundtrip 

time 

Client 

□ 

68.162.255.8 

16f2d58179d3de3d62b476f94cff7 

Mary  Player 

Online 

No 

connected 

0 

243 

& 

□ 

1198681139364 

WhoAgent2.csv 

Online 

No 

created 

0 

14 

Q 

[Participating  Players  (2) 


Figure  9  ELICIT  Dashboard  Screen 

As  part  of  creating  and  validating  this  new  agent-based  capability  a  stub  reference 
implementation  of  an  agent  was  developed. 

Note,  the  purpose  of  this  sample  stub  agent  is  to  demonstrate  an  agent  that  can  perform 
the  basic  actions  that  a  human  participant  can  make.  This  stub  agent  merely  performs 
these  basic  actions.  It  is  not  intelligent  enough  to  perform  them  in  an  intelligent  manner. 

Agent’s  actions  trigger: 

■  Click  Ready  button  after  configured  interval  after  a  trial  set  is  started.  Action 
is  triggered  by  receipt  of  a  New  Trial  Set  instruction  URL. 

■  Post  factoid  to  Who  website  after  configurable  delay  factor.  Action  is 
triggered  by  receipt  of  a  new  factoid  in  the  inbox. 

■  Pull  at  configurable  interval  from  a  specified  website  (Who). 

■  Share  factoid  after  configured  interval  after  factoid  is  received  in  inbox.  Share 
with  first  participant  on  Share  list. 

■  Move  factoid  to  MyFactoids  after  configured  interval  after  factoid  is 
received  in  inbox. 

■  Identify  action  after  configured  internal  after  factoid  is  received  in  inbox. 
Identify  content  is  hard  coded  (only  three  fields  are  provide). 

■  Once  received  by  the  Server  via  the  Agent  API,  the  server  treats  these  actions 
as  if  they  were  generated  by  a  human  participant. 


Additional  work  is  planned  to  develop  a  more  intelligent  sample  agent  and  eventually  to 
provide  more  guidance  to  researchers  who  wish  to  define  new  agents  (e.g  agents  with 
specific  personalities  or  intelligences.)  More  information  on  the  implementation  details 
of  the  new  agent-based  capability  is  available  in  the  Phase  2  -  Final  Detailed  Design  for 
Agent-based  Processing  vl.l  document. 
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Appendix  C  -  Experimentation  Campaign  Details 

In  this  appendix  we  elaborate  details  associated  with  the  experimentation  campaign  for 
reference.  As  in  the  body  of  the  paper  above,  the  campaign  framework  is  comprised  of 
four  dimensions:  1)  level  of  analysis,  2)  C2  approach,  3)  measure  of  effectiveness,  and  4) 
research  method.  We  summarize  these  dimensions  in  Table  1,  and  discuss  each  in  turn. 

Table  2  Campaign  Dimensions  (adapted  from  Nissen  2007) 


Campaign  Dimensions 


Level  of 

Analysis^ 

Independent^ 

(C2  Approach) 

Dependent^ 

(Measure) 

Research"^ 

(Method) 

Performance 

Analytical 

Social 

Organization 

Risk 

Computational 

Cognitive 

Process 

Agility 

Experimental 

Information 

People 

Learning 

Field 

Physical 

Technology 

Change 

Action 

1  Alberts  et  al.  (2001 )  2  Leavitt  (1 965)  3  CCRP  workshop  (2007)  4  Nissen  &  Buettner  (2004) 
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Level  of  analysis.  The  level  of  analysis  dimension  derives  from  Alberts  et  al.  (2001),  and 
includes  four  levels  relevant  to  C2  research:  1)  physical,  2)  information,  3)  cognitive  and 
4)  social.  First,  as  implied  by  the  name,  the  physical  level  involves  the  physical  hardware, 
software  and  equipment  associated  with  the  storage,  organization  and  transmission  of 
data  and  information.  This  would  include,  for  instance,  computers,  routers,  databases, 
cables,  and  wireless  transmission  and  reception  equipment.  Second,  and  implied  by  the 
name  likewise,  the  information  level  involves  data,  information  and  knowledge  that  flow 
through  organizations  This  would  be  enabled,  for  instance,  by  the  kinds  of  physical 
hardware,  software  and  equipment  noted  above,  and  would  include,  for  instance,  for 
instance,  the  kinds  of  data  stored  in  databases,  the  kinds  of  information  exchanged  via 
documents,  and  the  kinds  of  knowledge  learned  through  formal  courses  and  practical 
experience.  Both  the  physical  and  information  levels  pertain  to  inanimate  objects. 
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Third  and  alternatively,  the  eognitive  level  pertains  to  people,  and  involves  what 
people  feel,  know,  think  and  believe.  This  would  inelude,  for  instanee,  individual  sense 
making,  situational  awareness,  seareh,  analysis,  deeision  making,  and  like  eognitive 
proeesses.  This  would  be  enabled,  for  instanee,  by  the  kinds  of  data,  information  and 
knowledge  noted  above  that  flows  through  the  information  level,  and  whieh  is  enabled  in 
turn  by  the  kinds  of  physieal  hardware,  software  and  equipment  noted  above  in  the 
physieal  level.  Fourth  and  finally,  the  soeial  level  involves  interaetions  between 
individual  and  groups  of  people,  generally  viewed  within  an  organizational  eontext.  This 
would  inelude,  for  instance,  interpersonal  interactions  within  dyads,  groups, 
organizations  and  coalitions,  and  would  be  enabled  by  the  kinds  of  individual  sense 
making,  situational  awareness,  search,  analysis,  decision  making,  and  like  cognitive 
processes  noted  above.  Notice  how  each  level  is  enabled,  to  some  extent,  by  the  one 
below. 

C2  approach.  The  C2  approach  dimension  derives  from  Leavitt  (1965),  and  can  be 
thought  fruitfully  as  involving  the  independent  variables  that  are  selected  for  analysis  via 
an  experiment.  Leavitt  discusses  four  key  aspects  of  organizational  performance:  1) 
organization  structures,  2)  work  processes,  3)  people  and  4)  technologies.  Organization 
structures  involve  factors  such  as  the  allocation  of  decision  rights  and  job  responsibilities 
among  people  in  an  organization,  as  well  as  the  division  of  work  and  labor,  coordination 
mechanisms,  and  communication  and  work  routines.  Different  organization  structures 
(e.g..  Machine  Bureaucracy,  Simple  Structure;  see  Mintzberg  1979)  involve  different 
combinations  of  such  factors,  and  exert  dominant  effects  over  organizational 
performance.  Most  military  organizations,  for  instance,  are  very  hierarchical,  centralized 
and  bureaucratic,  and  hence  reflect  the  Machine  Bureaucracy  quite  closely.  As  suggested 
by  the  name,  work  processes  involve  the  arrangement  of  work  tasks  accomplished  within 
an  organization.  This  includes  task  sequencing,  timing  and  organization,  as  well  as  other 
aspects  of  the  work  itself  This  is  what  the  organization  does.  People  apply  their  skills  in 
organizations  to  perform  the  work  tasks.  They  reflect  different  education  and  training 
levels,  backgrounds,  cultures,  professions,  personalities  and  other  factors  associated  with 
the  diversity  of  people  working  in  organizations.  Finally,  technologies  involve  the  kinds 
of  physical,  electronic  and  manual  tools  used  by  people  to  perform  organizational  work. 
Most  such  tools  are  included  in  the  physical  level  described  above,  hence  one  can  see 
some  rough  correspondence  between  these  first  two  dimensions.  Leavitt  suggests  that 
these  four  factors  combine  to  outline  an  integrated  design,  and  that  all  four  should  be 
considered  together  whenever  addressing  organizational  performance  and  change.  When 
addressing  an  approach  to  C2,  all  four  factors  should  be  considered  together,  and  should 
complement  one  another  mutually. 

Measure  of  effectiveness.  The  measure  of  effectiveness  dimension  derives  from  CCRP 
(2007),  and  can  be  thought  fruitfully  as  involving  the  dependent  variables  that  are 
selected  for  analysis  via  an  experiment.  The  list  of  measures  can  be  very  long,  but  we 
have  identified  a  set  of  five  that  capture  the  most  pressing  aspects  of  C2  effectiveness 
today:  1)  performance,  2)  risk,  3)  agility,  4)  learning  and  5)  change.  Performance 
represents  the  most  common  measure  examined  via  C2  experimentation,  and  can  be 
unpacked  into  multiple  submeasures  (e.g.,  speed,  accuracy,  awareness).  Higher  levels  of 
performance  are  preferred  generally  to  lower  levels,  but  management  tradeoffs  exist 
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often  due  to  tensions  between  measures  (e.g.,  one  may  have  to  saerifiee  speed  for 
aecuracy).  Understanding  the  nature  of  sueh  tensions  via  experimentation  eontributes 
very  important  knowledge  for  C2.  Risk  is  viewed  by  many  (e.g.,  Turban  and  Aronson 
1998)  as  uneertain  events  to  whieh  probabilities  of  oecurrence  ean  be  assigned  reliably, 
but  we  expand  upon  this  relatively  narrow  view  to  include  conditions  of  uncertainty  (e.g., 
uncertain  events  to  which  probabilities  of  occurrence  cannot  be  assigned  reliably), 
equivocality  (e.g.,  contexts  in  which  the  important  factors  of  interest  are  not  understood 
well)  and  other  forms  of  ignorance  regarding  future  results.  Speaking  generally,  the 
greater  the  risk  associated  with  some  endeavor,  the  greater  the  likelihood  that 
performance  will  suffer.  Hence  risk  and  performance  appear  to  reflect  some  tension  and 
to  present  a  tradeoff  for  C2. 

Agility  is  discussed  by  Alberts  and  Hayes  (2003),  and  includes  six  aspects. 

Robustness  refers  to  the  ability  of  an  organization  to  perform  across  a  range  of  conditions 
(e.g.,  desert,  jungle,  urban).  Resilience  refers  to  the  ability  of  an  organization  to  recover 
from  or  adjust  easily  to  misfortune  or  damage.  Responsiveness  refers  to  the  ability  of  an 
organization  to  recognize  threats  and  opportunities,  and  to  take  the  appropriate  actions, 
quickly.  Flexibility  refers  to  the  ability  of  an  organization  to  shift  between  and  operate  in 
multiple  ways  to  adjust  to  changing  situations  and  contexts.  Innovation  refers  to  the 
ability  of  an  organization  to  learn  to  do  new  things,  and  to  learn  to  do  old  things  in  new 
ways.  Adaptivity  refers  to  the  ability  of  an  organization  to  change  its  C2  organization  and 
process.  Speaking  generally,  agility  represents  an  organization’s  approach  to  mitigating 
risk,  and  hence  is  related  to  performance  also.  However,  agility  introduces  its  own 
tension  with  performance,  as  agile  organizations  may  not  perform  as  well  in  some 
circumstances  as  their  more  rigid  but  finely  tuned  counterparts  do.  A  manager’s  focus  on 
agility  will  be  contingent  upon  the  nature  of  the  organization’s  particular  circumstances 
(see  Nissen  2005).  Learning  represents  the  change  in  performance  over  time,  and  hence 
can  be  used  to  describe  such  change  at  any  organizational  level  (e.g.,  individual,  group, 
organization).  Speaking  generally,  the  more  uncertain  and  dynamic  an  environment  is, 
the  more  important  that  learning  becomes.  Indeed,  a  manager  might  be  willing  to 
sacrifice  some  current  performance  in  order  to  gain  additional  learning.  Hence  learning 
and  performance  involve  some  tension  and  corresponding  managerial  tradeoffs  as  well. 
Finally,  change  represents  the  ability  to  alter  one’s  structure  and  behavior  over  time.  Like 
learning,  this  can  be  used  to  describe  such  change  at  any  organizational  level  (e.g., 
individual,  group,  organization).  Speaking  generally,  the  more  uncertain  and  dynamic  an 
environment  is,  the  more  important  that  change  becomes.  However,  change  involves 
disruption  to  performance  inevitably,  and  hence  involves  some  tension  and  corresponding 
managerial  tradeoffs  as  well.  One  can  see  that  these  five  measures  of  effectiveness  are 
interrelated  richly.  Examination  the  various  tensions  and  managerial  tradeoffs  between 
them  represents  an  important  aspect  of  C2  experimentation. 

Research  method.  Research  method  derives  from  Nissen  and  Buettner  (2004),  and  refers 
to  the  approach  that  an  experimenter  will  take  to  developing  new  C2  knowledge.  This 
dimension  lies  somewhat  outside  of  the  framework  for  describing  experimentation 
campaigns,  for  laboratory  experimentation  represents  one  of  several  alternate  research 
methods  that  can  be  employed.  The  five  methods  associated  with  this  framework  are 
interrelated  closely,  however,  in  that  they  form  something  of  a  sequence  or  life  cycle  that 
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ean  be  used  for  planning  a  researeh  campaign:  1)  analytical  methods,  2)  computational 
methods,  3)  laboratory  experimentation,  4)  field  research  and  5)  action  research.  Each 
research  method  has  distinct  advantages  and  disadvantages  with  respect  to  others.  We 
consider  each  in  turn. 

Eirst,  we  have  analytical  methods,  which  can  be  performed  generally  within  a 
researcher’s  office,  with  or  without  the  aid  of  computers  or  other  office  equipment. 
Mathematical  modeling,  regression  analysis  and  optimization  represent  three  instances  of 
analytical  methods.  Many  research  campaigns  begin  with  analytical  methods,  for  they  are 
relatively  inexpensive  and  quick  to  employ,  and  research  results  generated  through  such 
methods  can  provide  useful  guidance  to  inform  research  conducted  via  other  methods 
(esp.  computational  methods).  Eor  instance,  mathematical  modeling  might  suggest  a 
limited  set  of  variables  and  relations  with  potential  to  influence  C2  performance,  risk  and 
other  measures  of  effectiveness.  Such  variables  and  relations  could  be  examined  with 
greater  fidelity  via  computational  methods,  for  instance.  However,  concepts,  variables 
and  relationships  are  specified  abstractly  for  the  most  part  in  analytical  methods,  and 
generalizing  the  results  of  laboratory  experiments  to  operational  organizations,  tasks, 
people  and  technologies  outside  the  laboratory  can  be  very  difficult. 

Second,  we  have  computational  methods,  which  can  be  performed  generally  with  a 
researcher’s  office  also,  but  which  depend  expressly  upon  computer  hardware  and 
software.  Indeed,  to  the  extent  that  a  researcher  uses  computational  tools  to  represent 
mathematical  models,  to  conduct  regression  analyses  or  to  solve  optimization  problems, 
computational  methods  and  analytical  method  appear  to  converge.  However,  many 
problems  cannot  be  solved  via  mathematical  models,  regression  analyses,  optimization 
problems,  or  other  analytical  methods,  and  hence  must  rely  upon  other  computational 
methods.  Simulation  represents  a  very  common  instance  of  computational  research 
method,  and  simulation  models  can  be  run  in  a  controlled  manner  reflecting  laboratory 
experimentation  very  closely;  such  simulation  approach  is  referred  to  as  computational 
experimentation.  As  noted  above  for  analytical  methods,  computational  methods  are 
relatively  inexpensive  and  quick  to  employ  also,  and  research  results  generated  through 
such  methods  can  provide  useful  guidance  to  inform  research  conducted  via  other 
methods  (esp.  laboratory  experimentation).  Eor  instance,  computational  experimentation 
might  suggest  a  limited  set  of  variables  and  relations  with  potential  to  influence  C2 
performance,  risk  and  other  measures  of  effectiveness.  Such  variables  and  relations  could 
be  examined  with  greater  realism  via  laboratory  experimentation,  for  instance. 

Third,  we  have  laboratory  experimentation.  This  is  the  central  research  method 
associated  with  an  experimentation  campaign,  and  involves  the  study  of  people  and 
equipment  within  the  laboratory  under  controlled  conditions.  Eaboratory  experimentation 
enables  tremendous  control  over  the  environment,  and  results  reflect  great  reliability  and 
internal  validity  generally.  However,  laboratory  experimentation  tends  to  be  more 
expensive  and  require  more  time  than  either  analytical  or  computational  methods,  and 
generalizing  the  results  of  laboratory  experiments  to  operational  organizations,  tasks, 
people  and  technologies  outside  the  laboratory  can  be  difficult  often.  EEICIT  is  a  tool  for 
laboratory  experimentation.  Research  results  generated  through  such  methods  can 
provide  useful  guidance  to  inform  research  conducted  via  other  methods  (esp.  field 
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research).  For  instance,  laboratory  experimentation  might  suggest  a  limited  set  of 
variables  and  relations  with  potential  to  influence  C2  performance,  risk  and  other 
measures  of  effectiveness.  Such  variables  and  relations  could  be  examined  with  greater 
realism  via  field  research,  for  instance. 

Fourth,  we  have  field  research.  This  involves  the  study  of  C2  organizations,  work 
processes,  people  and  technologies  in  their  natural  environments  (i.e.,  in  the  field).  A 
wide  variety  of  field  research  methods  are  available,  but  case  studies,  ethnographies  and 
field  experiments  are  common  in  the  C2  domain.  Field  studies  have  different  properties 
than  laboratory  experiments  do.  They  enable  negligible  control  over  the  environment, 
hence  results  reflect  poor  reliability  and  internal  validity  generally,  and  field  research 
tends  to  be  much  more  expensive  and  require  much  more  time  than  laboratory 
experimentation  does.  However,  generalizing  the  results  of  field  research  to  operational 
organizations,  tasks,  people  and  technologies  is  straightforward  generally,  and  the  realism 
of  field  research  represents  a  strength  of  the  method.  Research  results  generated  through 
such  methods  can  provide  useful  guidance  to  inform  research  conducted  via  other 
methods  (esp.  action  research).  For  instance,  field  research  might  suggest  a  limited  set  of 
variables  and  relations  with  potential  to  influence  C2  performance,  risk  and  other 
measures  of  effectiveness.  Such  variables  and  relations  could  be  examined  with  greater 
realism  via  action  research,  for  instance. 

Finally,  we  have  action  research.  This  involves  working  to  make  positive  changes  in 
operational  organizations,  tasks,  people  and  technologies,  but  seeking  to  learn  through 
systematic  study  at  the  same  time.  Very  similar  to  its  field  research  counterpart,  action 
research  methods  enable  negligible  control  over  the  environment,  hence  results  reflect 
poor  reliability  and  internal  validity  generally,  and  action  research  tends  to  be  much  more 
expensive  and  require  much  more  time  than  laboratory  experimentation  does.  However, 
generalizing  the  results  of  action  research  to  operational  organizations,  tasks,  people  and 
technologies  is  straightforward  generally,  and  the  realism  of  action  research  in 
unparalleled.  Research  results  generated  through  such  methods  can  have  immediate  and 
practical  impact.  This  represents  a  strength  of  the  method. 
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•  ELICIT  Introduction 

•  New  Capabilities 

—Configurable  Organizations 
-Agent-based  processing 

•  Experimentation  Campaign 


Overview 
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•  ELICIT-Experimental  Laboratory  for 
Investigating  Collaboration, 
Information-sharing,  and  Trust 

—Part  of  CCRP's  network-centric  warfare 
initiative 

—Developing  and  testing  principles  of 
organization  that  transfer  power  and 
decision  rights  to  the  edge 

—Testing  hypotheses  through  experiments 

—Facility  was  created  to  run  experiments 


ELICIT  Introduction 
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•  Identify  the  who,  what,  where  and 
when  of  an  adversary  attack 

•  Task  scenarios  are  anonymized 

•  Participants  are  anonymized 

•  Participants  periodically  receive 
factoids  about  the  situation 

•  No  one  person  has  all  information 

•  Task  success  requires  communication 

•  All  communication  via  software 
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•  Original  organization  modeling 

•  Motivation  for  enhancement 

•  New  flexible  organization  modeling 
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Original  Organization 

Mndplinn 
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•  Execution  flexibility  -  I  don't  have  17 
subjects  today 

•  Modeling  hybrid  organizations 

•  Modeling  participant  organizations 

•  Modeling  availability  of  new  capabilities 


Motivation  for  Enhancement 
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How  to  read  table: 

n|Role|Teann|Country|  1|2|3|4|5|6|7|8|9|  10|  11|  12|  13|  141 15|  16|  17|websitel|website2|website31websi 
te4| 

<begin  actual  table> 

Cross  Team  Coordinator|Chiland |0|1|1|1|1|1|1|1|1|1|1|111|1|1|1|1|1|1|1|1| 


1 

2|Team  leader 
3jTeam  leader 
4jTeam  leader 
sjTeam  leader 
6jTeam  member 
7jTeam  member 
sjTeam  member 
9jTeam  member 
10|Team  member 
lljTeam  member 
12jTeam  member 
13iTeam  member 
14jTeam  member 
IsjTeam  member 
IsjTeam  member 
17jTeam  member 
B 

Who 

what 

where 

when 


|Who|Chiland|l|0|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|0|0|0| 

|what|Psiland|lll|0|l|l|l|l|l|l|l|l|l|l|l|l|l|l|0|l|0|0| 

|where|Omegalandll|l|l|0|l|l|l|l|l|l|l|l|l|l|l|l|l|0|0|l|0| 

|when|Deltaland|lll|l|l|0|l|l|l|l|l|l|l|l|l|l|l|l|0|0|0|l| 

|Who|Chiland|l|l|l|l|l|0|l|l|l|l|l|l|l|l|l|l|l|l|0|0|0| 

|Who|Chiland|l|l|l|l|l|l|0|l|l|l|l|l|l|l|l|l|l|l|0|0|0| 

|who|Chilandll|l|l|l|l|l|l|0|l|l|l|l|l|l|l|l|l|l|0|0|0| 

|what|Psilandll|l|l|l|l|l|l|l|0|l|l|l|l|l|l|l|l|0|l|0|0| 

|what|Psilandll|l|l|l|l|l|l|l|l|0|l|l|l|l|l|l|l|0|l|0|0| 

|what|Psilandll|l|l|l|l|l|l|l|l|l|0|l|l|l|l|l|l|0|l|0|0| 

|where|Omegaland|l|l|l|l|l|l|l|l|l|l|l|0|l|l|l|l|l|0|0|l|0| 

|where|Omegalandll|l|l|l|l|l|l|l|l|l|l|l|0|l|l|l|l|0|0|l|0| 

|where|Omegaland|l|l|l|l|l|l|l|l|l|l|l|l|l|0|l|l|l|0|0|l|0| 

|when|Deltalandll|l|l|l|l|l|l|l|l|l|l|l|l|l|0|l|l|0|0|0|l| 

|when|Deltalandll|l|l|l|l|l|l|l|l|l|l|l|l|l|l|0|l|0|0|0|l| 

|when|Deltalandll|l|l|l|l|l|l|l|l|l|l|l|l|l|l|l|0|0|0|0|l| 
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Loading  Organization  File 


Configuration  Dashboard  Console  Logs  Config  Files  Agent  Registration 


Configuration  files 


Select  configuration  file  to  upload  onto  server. 

File  name  should  begin  with  factoidset,  names,  countries  or  organization  and  ends  with  .csv 

ELICIT  Configuration  Agent  Configuration  Agent  JAR  File 

0  O  O 

File  name 


Browse... 


Upload  I  I  Reset 


Factoid  Sets 

Names 

Countries 

Orq 

anizations 

Aaent  Conf  Files 

Aaent  JAR  Files  1 

Del 

Del 

Del 

Del 

Del 

Del 

□ 

factoidset5-17.csv 

□ 

namesl7.csv 

□ 

countriesl.csv 

□ 

organizationTypeC2-17.csv 

□ 

WhoAgent.csv 

□ 

'WhoAqent.iari 

□ 

factoidsetS-B.csv 

□ 

countries4.csv 

□ 

organizationE C2 mixed-17.csv 

□ 

WhoAgent2.csv 

□ 

factoidset4-8.csv 

□ 

countries3.csv 

□ 

organizationE-17.csv 

□ 

factoidset4-l.csv 

□ 

countries2.csv 

□ 

factoidsetl-17.csv 

□ 

factoidset4-17.csv 

□ 

factoidset3-17.csv 

□ 

factoidsetl-5.csv 

□ 

factoidset4-2.csv 

□ 

factoidsetl-l.csv 

□ 

factoidset4-5.csv 

□ 

factoidset2-17  .CSV 

iFactoidset  Files  ( 12)|Name  Files  (1)  ICountries  Files  (4)|Orqanization  Files  (3)  lAqent  Conf  Files  f2MAqent  JAR  Files  (1)1 
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•  Motivation  for  enhancement 

•  New  capability 

•  Ongoing  agent  development 
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•  Flexibility  in  staffing  experiments 
(hybrid  runs) 

•  Ability  to  run  many  more  experiments 

•  Theories  of  behavior  and  performance 
could  be  validated  by  constructing 
agents  with  specific  capabilities 
(personalities) 
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Initial  Phase 

•  Agent  harness  added 

•  Stub  reference  agent  created  to 
validate  harness 

Current  Phase 

•  Agent  batch  capability 

•  Sense  Making,  moderately  intelligent 
agent  under  development 
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•  Moderately  intelligent 

•  Is  aware  of  aspects  of  the  factoids  that 
have  been  presented  to  it 

•  Builds  a  mental  model  of  what  it  knows 

•  Models  decision  making  and  human 
errors 

•  Supports  some  aspects  of  personas 

(around  sharing,  etc.) . 

•  Work  is  on  going. .. _ 
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Dependent 

(Measure) 


Level  of  Analysis  -Action 

P  A  R  II  T  Y  80  unique  study  areas  (4x4x5  =  80)  x  5  methods  14 
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•  Raw  data  and  materials  posted  to 
http://www.dodccrp.orq/html4/elicit.ht 

ml 

•  For  more  information  contact 

-Dr.  David  Alberts,  NII/DoD-CIO 
David  ■Alberts@osd.  mil 

—Dr.  Mark  Nissen,  MNissen@nps.edu 

—Mary  Ruddy  marv@paritvinc.net 
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Questions  ? 
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